<globalJmxStatistics enabled="true"/>
Management of Infinispan instances is all about exposing as much relevant statistical information that allows administrators to get view of the state of each Infinispan instance. Taking in account that a single installation could be made up of several tens or hundreds Infinispan instances, providing clear and concise information in an efficient manner is imperative. The following sections dive into the range of management tooling that Infinispan provides.
Over the years, JMX has become the de facto standard for management and administration of middleware and as a result, the Infinispan team has decided to standarize on this technology for the exposure of management or statistical information.
JMX reporting can be enabled at 2 different levels:
1. CacheManager level: The CacheManager is the entity that governs all the cache instances that have been created from it. Details on the information exposed at the CacheManager level can be found below. For the moment, let's just focus on how to enable the CacheManager to report management data via JMX.
If configuring the CacheManager via XML, make sure you add the following XML under the <global> element:
<globalJmxStatistics enabled="true"/>
If configuring the CacheManager programmatically, simply add the following code:
GlobalConfiguration globalConfiguration = ... globalConfiguration.setExposeGlobalJmxStatistics(true);
2. Cache level: At this level, you will receive management information generated by individual cache instances. Details on the information exposed at the Cache level are explained below. For the moment, let's just focus on how to enable the Cache to report management data via JMX:
If configuring the Cache via XML, make sure you add the following XML under the either <default>, if you're configuring the default Cache instance, or under the corresponding <namedCache>:
<jmxStatistics enabled="true"/>
If configuring the Cache programmatically, simply add the following code:
Configuration configuration = ... configuration.setExposeJmxStatistics(true);
Once you have enabled JMX reporting at either the CacheManager or Cache level, you should be able to connect to VM(s) where Infinispan is running using a standard JMX GUI such as JConsole or VisualVM, and you should find the following MBeans:
If you enabled CacheManager level JMX statistics, you should see an MBean called infinispan:cache-name=[global],jmx-resource=CacheManager with properties specified by the CacheManager MBean.
If you enabled Cache level JMX statistics, you should see several different MBeans depending on which configuration options have been enabled. For example, if you have configured a write behind cache store, you should see an MBean exposing properties belonging to the cache store component. All Cache level MBeans will follow the same format though which is the following: infinispan:cache-name=<name-of-cache>(<cache-mode>),jmx-resource=<component-name> where:
<name-of-cache> has been substituted by the actual cache name. If this cache represents the default cache, its name will be "___defaultcache".
<cache mode> has been substituted by the cache mode of the cache. The cache mode is represented by the lower case version of the possible enumeration values shown here.
<component-name> has been substituted by one of the JMX component names in the JMX reference documentation.
For example, the cache store JMX component MBean for a default cache configured with synchronous distribution would have the following name:infinispan:cache-name=___defaultcache(dist_sync),jmx-resource=CacheStore
Please note that any cache names that contain ':' or '=' characters will be substituted by '_' character. Infinispan does this because ':' and '=' are control characters for JMX object names.
If you enabled CacheManager level JMX statistics, without further configuration, you should see an MBean called org.infinispan:type=CacheManager,name="DefaultCacheManager" with properties specified by the CacheManager MBean.
Using the cacheManagerName attribute in globalJmxStatistics XML element, or using the corresponding GlobalConfiguration.setCacheManagerName() call, you can name the cache manager in such way that the name is used as part of the JMX object name. So, if the name had been "Hibernate2LC", the JMX name for the cache manager would have been: org.infinispan:type=CacheManager,name="Hibernate2LC" . This offers a nice and clean way to manage environments where multiple cache managers are deployed, which follows JMX best practices.
If you enabled Cache level JMX statistics, you should see several different MBeans depending on which configuration options have been enabled. For example, if you have configured a write behind cache store, you should see an MBean exposing properties belonging to the cache store component. All Cache level MBeans follow the same format though which is the following: org.infinispan:type=Cache,name="<name-of-cache>(<cache-mode>)",manager="<name-of-cache-manager>",component=<component-name> where:
<name-of-cache> has been substituted by the actual cache name. If this cache represents the default cache, it's name will be "_defaultCache" .
<cache-mode> has been substituted by the cache mode of the cache. The cache mode is represented by the lower case version of the possible enumeration values shown here.
<name-of-cache-manager> has been substituted by the name of the cache manager to which this cache belongs. The name is derived from the cacheManagerName attribute value in globalJmxStatistics element.
<component-name> has been substituted by one of the JMX component names in the JMX reference documentation.
For example, the cache store JMX component MBean for a default cache configured with synchronous distribution would have the following name: org.infinispan:type=Cache,name="_defaultcache(dist_sync)", manager="DefaultCacheManager",component=CacheStore
Please note that cache and cache manager names are quoted to protect against illegal characters being used in these user-defined names.
There can be situations where several CacheManager instances are created in a single VM, or Cache names belonging to different CacheManagers under the same VM clash.
In order to cope with such situations, Infinispan enabled users to define a particular JMX domain prefix for their MBeans. For example, either of these two options could be used to configure a JMX domain prefix called "myInfinispan" for the CacheManager JMX statistics:
Via XML:
<globalJmxStatistics enabled="true" jmxDomain="myInfinispan"/>
Programmatically:
GlobalConfiguration globalConfiguration = ...
globalConfiguration.setExposeGlobalJmxStatistics(true);
globalConfiguration.setJmxDomain("myInfinispan");
Using either of these options should result on the CacheManager MBean name being: myInfinispan:cache-name=[global],jmx-resource=CacheManager
As you probably have guessed by now, the default JMX domain prefix is called "infinispan ".
Another related configuration option for both CacheManager and Cache JMX statistics allows for duplicate JMX domains to be discovered. Internally, when duplicates are allowed, Infinispan takes the duplicating JMX domain prefix, adds an index that starts at number 2 to the existing prefix and uses that JMX prefix from then onwards. So, for example, if two CacheManagers were started with global JMX statistics enabled, no particular JMX domain was configured, and JMX domain duplicates were allowed, the first CacheManager would be registered under "infinispan... ", whereas the second one would be registered under: "infinispan2... ". To allow JMX duplicate domains, do the following:
Via XML:
<globalJmxStatistics enabled="true" allowDuplicateDomains="true"/>
Programmatically:
GlobalConfiguration globalConfiguration = ... globalConfiguration.setExposeGlobalJmxStatistics(true); globalConfiguration.setAllowDuplicateDomains(true)
Remember that by default, duplicate domains are disallowed.
Using different JMX domains for multi cache manager environments should be last resort. Instead, as mentioned in previous section, it's now possible to name a cache manager in such way that it can easily be identified and used by monitoring tools such as RHQ. For example:
Via XML:
<globalJmxStatistics enabled="true" cacheManagerName="Hibernate2LC"/>
Programmatically:
GlobalConfiguration globalConfiguration = ...
globalConfiguration.setExposeGlobalJmxStatistics(true);
globalConfiguration.setCacheManagerName("Hibernate2LC");
Using either of these options should result on the CacheManager MBean name being: org.infinispan:type=CacheManager,name="Hibernate2LC"
Please note as well that since 4.2, the default domain names has changed from "infinispan" to "org.infinispan", as per JMX best practices.
For the time being, you can still set your own jmxDomain if you need to and we also allow duplicate domains, or rather duplicate JMX names, but these should be limited to very special cases where different cache managers within the same JVM are named equally.
To finish up with this JMX section, let's quickly discuss where Infinispan registers all these MBeans. By default, Infinispan registers them in the standard JVM MBeanServer plattform. However, users might want to register these MBeans in a different MBeanServer instance. For example, an application server might work with a different MBeanServer instance to the default plattform one. In such cases, users should implement the MBeanServerLookup interface provided by Infinispan so that the getMBeanServer() method returns the MBeanServer under which Infinispan should register the management MBeans. You can find an example in the default PlatformMBeanServerLookup class used by Infinispan. So, once you have your implementation ready, simply configure Infinispan with the fully qualified name of this class. For example:
Via XML:
<globalJmxStatistics enabled="true" mBeanServerLookup="com.acme.MyMBeanServerLookup"/>
Programmatically:
GlobalConfiguration globalConfiguration = ...
globalConfiguration.setExposeGlobalJmxStatistics(true);
globalConfiguration.setMBeanServerLookup("com.acme.MyMBeanServerLookup")
There has been a couple of noticeable additions in Infinispan 5.0 in terms of MBean exposed:
MBeans related to Infinispan servers are now available that for the moment focus on the transport layer. So, if the Infinispan servers are configured with global JMX statistics, a brand new MBean in org.infinispan:type=Server,name=<Memcached|Hotrod>,component=Transport is now available which offers information such as: host name, port, bytes read, byte written, number of worker threads...etc.
When global JMX statistics are enabled, JGroups MBeans are also registered automatically, so you can get key information of the group communication transport layer that's used to cluster Infinispan instances. To find out more about the information provided, check the JGroups JMX documentation.
The preferred way to manage multiple Infinispan instances spread accross different servers is to use RHQ, which is JBoss' enterprise management solution. Thanks to RHQ's agent and auto discovery capabilities, monitoring both Cache Manager and Cache instances is a very simple task. With RHQ, administrators have access to graphical views of key runtime parameters or statistics and can also be notified be these exceed or go below certain limits. The Infinispan specific statistics shown by RHQ are a reflection of the JMX information exposed by Infinispan which has been formatted for consumption by RHQ. Please follow these steps to get started with RHQ and have Infinispan instances monitored with it:
Firstly, download and install an RHQ server and install and start at least one RHQ agent. The job of the RHQ agent is to send information about the Infinispan instance back to the server which is the one that shows the information via a nice GUI. You can find detailed information on the installation process in RHQ's installation guide and you can find information on how to run an agent in the RHQ agent guide.
If you're just building a demo or testing RHQ server, you can avoid the need to install a fully fledged database and use an in-memory H2 database instead. However, you might encounter issues after testing database connection as shown here. Simply repeating the installation avoiding testing the connection should work.
The most common set up is to have the RHQ agent installed in the same machine where Infinispan is running. If you have multiple machines, an agent can be installed in each machine.
By now, you should have an RHQ server and agent running. It's time now to download the latest Infinispan binary distribution (*-bin.zip or *-all.zip should do) from the downloads section and locate the RHQ plugin jar file which should be named something like infinispan-rhq-plugin.jar. This is located under the modules/rhq-plugin directory.
The adding and updating plugins section on the RHQ guide contains some more detailed information on how to update both RHQ servers and agents with new plugins, but essentially, this process involves uploading a new plugin to the RHQ server and then pushing the plugin to one, or several, RHQ agents.
If you're simply demoing or testing and you only have a single agent, once the plugin has been uploaded to the server, simply go to the agent command line interface and type: plugins update .This will force the agent to retrieve the latest plugins from the server. Doing this can be considerably faster than some of the other alternatives.
At this point, RHQ is ready to start monitoring Infinispan instances, but before firing them up, make sure you start them with the following system properties so that RHQ agents can discover them:
-Dcom.sun.management.jmxremote.port=6996 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
The actual port value used does not really matter here, but what matters is that a port is given, otherwise Infinispan instances cannot be located. So, you can easily start multiple Infinispan instances in a single machine, each with a different remote JMX port, and a locally running agent will be able to discover them all without any problems.
Once Infinispan instances have been discovered, you should see a new resource for each of the cache manager running appearing in the Inventory/Discovery Queue of the RHQ server. Simply import it now and you should see each cache manager appearing with as many child cache resources as caches are running in each cache manager. You're now ready to monitor Infinispan!
This section focuses on the lessons learned while developing the Infinispan RHQ plugin that are likely to be useful to anyone using RHQ.
By default, at least in version 2.3.1 of RHQ, the RHQ agent sends an availability report of any managed resources every 5 minutes. The problem with this is that if you're testing whether your Infinispan instance is automatically discovered by the RHQ server, it can take up to 5 minutes to do so! Also, it can take 5 minutes for the RHQ server to figure out that you've shutdown your Infinispan instance. You can change this setting by the following property (default value is 300 seconds) in rhq-agent/conf/agent-configuration.xml. For example, if you wanted the availability to be sent every 1 minute, simply change the value to 60:
<entry key="rhq.agent.plugins.availability-scan.period-secs" value="60"/>
Please bear in mind the instructions given in the RHQ agent installation and more specifically the paragraph below with regards to changes made to properties in agent-configuration.xml:
Once the agent is configured, it persists its configuration in the Java Preferences backing store. Once this happens, agent-configuration.xml is no longer needed or used. Editing agent-configuration.xml will no longer have any effect on the agent, even if you restart the agent. If you want the agent to pick up changes you make to agent-configuration.xml, you must either restart the agent with the "--cleanconfig" command line option or use the "config --import" agent prompt command.
As mentioned in the previous section, RHQ consumes the JMX data exposed by Infinispan, and in similar fashion, plugins could be written for other 3rd party management tools that were able to transform these data into the correct representation in these tools, for example graphs,...etc.